Papers with interpretable evaluation framework
The olmOCR Project: Building Fully Open OCR using VLMs (2026.acl-demo)
Copied to clipboard
| Challenge: | olmOCR is a fully open OCR system developed through iterative public releases and community feedback. |
| Approach: | They propose an open OCR system that combines a 7B vision-language model trained in two stages: finetuning and reinforcement learning with visual unit tests. |
| Outcome: | The proposed system achieves state-of-the-art performance among open systems and proprietary APIs at a fraction of the cost. |
Learning Personalized Alignment for Evaluating Open-ended Text Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Traditional evaluation metrics rely heavily on lexical similarity with human-written references, showing poor correlation with human judgments and failing to account for alignment with the diversity of human preferences. |
| Approach: | They propose an interpretable evaluation framework that evaluates alignment with specific human preferences by providing detailed comments and fine-grained scoring. |
| Outcome: | The proposed framework outperforms GPT-4 in Kendall correlation and accuracy with zero-shot reviewers. |
Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that MT metrics return assessments as scalar scores that are difficult to interpret, posing a challenge to making informed design choices. |
| Approach: | They propose an interpretable evaluation framework that evaluates MT metrics in two scenarios that serve as proxies for filtering and translation re-ranking use cases. |
| Outcome: | The proposed framework offers clearer insights than correlation with human judgments. |